1. Field of the Invention
The present invention relates to systems and methods for assessing call center performance and, in particular, to systems and methods that allow the performance and current and potential levels of automation of a call center to be quantified.
2. Discussion of the Related Art
Telephone user interfaces are the most widespread class of human-computer interfaces. Introduced more than a decade ago, touch-tone interactive voice response (IVR) systems were adopted enthusiastically in many call-centers and promised to provide customer service efficiently. However, calling customers (callers) have exhibited antipathy towards touch-tone IVR systems, viewing them as difficult to use. Further aggravating the callers is the fact that they often endure long waiting times before they can speak to live agents. This dichotomy is not surprising considering that most call-centers focus on minimizing operating costs and that usability and its impact on call-center operations are poorly understood.
Since touch-tone IVR systems have been deployed for more than a decade, a significant body of know-how on IVR systems has accumulated in the industry. Except for recent attempts to define a style guide for (telephone) speech applications, as in Balentine, B. and D. P. Morgan, How to Build A Speech Recognition Application, 1999, San Ramon, Calif., Enterprise Integration Group, and to introduce universal commands in speech-enabled IVR systems, as in Cohen, M., Universal Command for Telephony-Based Spoken Language Systems, SIG-CHI Bulletin, 2000, 32(2), pp. 25-30, this body of knowledge is not well documented. The prevalence of usability problems in deployed IVR systems suggests that designing good telephone interfaces is difficult and usability engineering methods for telephone interfaces are not well developed.
Another area of related work is research on spoken dialog systems, an important application of speech recognition technology. Spoken dialog systems allow the caller to communicate with a system in a spoken dialog, not necessarily over the telephone. Many research articles on spoken dialog systems have been published, e.g., Stallard, D., Talk'N'Travel: A Conversational System for Air Travel Planning, in Applied Natural Language Processing ANLP, 2000, Seattle, Wash.; Peckham, J., A new generation of spoken language systems: recent results and lessons from the SUNDIAL project, in European Conference on Speech Communication and Technology EUROSPEECH, 1993, Berlin (Germany): European Speech Communication Association; Levin, E. and R. Pieraccini, CHRONUS: The Next Generation, in ARPA Workshop on Spoken Language Technology, 1995, Austin (TX): Morgan Kaufmann Publishers, Inc.; Bennacef, S., et al., Dialog in the RIALTEL telephone-based system, in International Conference on Spoken Language Systems ICSLP, 1996, Philadelphia, Pa.; and Lee, C. H., et al., On Natural Language Call Routing, in Speech Communications, 2000, 31, pp. 309-320.
Previous work on spoken-dialog system evaluation focused on quantifying the performance of the underlying technologies, e.g., Chang, H., A. Smith, and G. Vysotsky, An automated performance evaluation system for speech recognizers used in the telephone network, in International Conference on World Prosperity Through Communications, 1989; and Pallett, D. S., et al., 1993 Benchmark Tests for the ARPA Spoken Language Program, in ARPA Workshop on Spoken Language Technology, 1994, Princeton (NJ): Morgan Kaufmann Publishers, Inc.
Some studies have evaluated the usability of telephone interfaces based on task completion rates and post-experimental questionnaires, e.g., Edwards, K., et al., Evaluating Commercial Speech Recognition and DTMF Technology for Automated Telephone Banking Services, in IEEE Colloquium on Advances in Interactive Voice Technologies for Telecommunication Services, 1997. More recently, PARADISE was introduced as a “consistent integrative framework for evaluation” of spoken language systems, as described in Walker, M. A., et al., PARADISE: A Framework for evaluating spoken dialogue agents, in 35th Annual Meeting of the Association of Computational Linguistics, 1997, Madrid: Morgan Kaufmann Publishers, Inc. Basically, PARADISE provides a method to identify measures that predict user satisfaction well, from the large set of measures that have been used in the field. However, this work does not address the cost for the call center, nor does it provide any guidance for telephone interface redesign.